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ABSTRACT 

In order to benefit fran e?q)erience gained from the 

synchronization of operational flddeband digital net- ^ 

works, a survey was made of three such systems: 

Data Ti’ananission Company, Western IfeicHi Tel^raph 
Conpany, and the Conputer Comnunications Groi^) of 
the Trans-Canada Telephcme System. Additional infor- 
mation was obtained from AT&T relative to their 
Switched Digital Network. 

The focus of the survey was on deployment and opera- 
tional esqjerience from a practical (as exposed to 
theoretical) viewpoint. The objective was to pro- 
vide a report on the results of deployment-how the 
systems perfoimed and vdierein the performance 
differed from that predicted or intended in the 
design. It also attenpted to determine how the 
varioxis system designers would use the benefit of 
hindsight if they could design those same systems 
today. 

No conclusions or evaluations of the network per- 
formance are provided in the report. However, 
some of the differences in requirements between the 
cormercial networks surveyed and those of strategic, 
survivable networks, such as the Defense Coninunica- 
tions System, are noted. 

IjTRODUCTiqj 

There are (were) several sideband digital communications networks 
operating which incorporate synchronization systems. Most of these net- 
works were originally conceived and developed in the late sixties and 
early seventies, and initially deployed in the early seventies. This 
paper describes the results of a survey which attempted to learn of 
practical aspects and experiences in the design, deployment and operation 
of these networks. 

Three specific systems surveyed are the switched digital network 
developed by the Data Transmission Conpany and now operated by SP Cem- 
munications (hereafter called the Datran System); the digital network 
deployed and operated by the Computer Communications Groip of the Trans- 
Canada Telephone System (Dataroute) and a digital system developed by the 
Western Itolon Tel^aph Co. (hereafter called the WU system). With the 
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exception of the WU System, these networks are in service today. The 
Western Uniaa system was put into (^ration in late 1971 and field trials 
conducted for several months. The system was never put into service, 
and has since been dismantled. 

A brief descriptican of each major subsystem of the three networks is 
given. It is follov^ by a set of questions vrtiich were posed to per- 
sonr'^1 associated with each of the networks in the development and 
early operational pheises. These questions were posed during meeti-ngs 
with the personnel for each of the systems. Original plans were to in- 
clude a section on the private-line Digital Data Systan (IX)S) deployed 
by American Telephone and Telegraph. However, due to pending litiga- 
tion, AT&T personnel were unable to provide information on EOS. An in- 
terview was substituted which covered Bell's planning for timing and 
synchronization of the net>?ork of No. 4ESS switching centers referred 
to as the Switched Data Network (SKI) . 

The pap)er concludes with a ' .imary of the survey and a general com- 
parison of timing/synchronization (T/S) considerations for the coimer- 
cial networks described with those for a strategic military network 
such as the EOS. Many of the features inportant to a military system 
such as the DCS need not be considered in a cotmercial system. Survi- 
vability may be of interest to a coimercial network, but not enough to 
dominate design trade-offs. Thus, while the engineering, planning, and 
design of the conitiercial networks, along with the operational experien- 
ces, are of interest in planning military networks, the two applica- 
tions have distinctions that can heavily sway design trade-offs and 
decisions in different vrays. 

Questions posed to the technical pjersonnel associated with the three 
digital systems (WU, Datran, Eteitaroute) were as follows: 

A. Fundamental Reasons for Choosing Timing and Synchronization 

(T/S) As Inplemented. 

1. What were criteria used for selecting T/S system? 

2. Were other types T/S system considered? 

3- On what basis was final choice made-p)erfontBnce analysis; 
cost analysis; 0/M analysis; simulation; other? 

4. Is system reference to UTC or other standard necessary and/ 
or desirable? 

5. How was OGnpatibility with other systems (international, 
military, other common carriers, etc.) a factor in the sys- 
tem selection/design? 

B. Theoretical Design of Chosen T/S System 

1. What were the main criteria in sp>ecifying the T/S system? 

Did it have to "marry" existing hardware, etc.? 

2. Spjecifications of Hardware 




(a) Stability - long/short term 

(b) Reliability 

(c) Jitter reduction 

(d) Cost goals 

(e) 0/M aspects 

3. What were driving criteria on timing distribution waveforms, 
interfaces, etc. "downstream" frcm clocks, i.e., between 
MUX's, from MUX's to user, etc.? 

C. Practical Design/Engineering Considerations 

1. Were any unusual obstacles met in the design/developinent of 
hardware? 

2. Were specifications as finally met tighter or looser than 
original specifications? 

D. Initial Field Eteployment, Testing, O&M 

1. In the field trials and early operation of the network, were 
any unusual problons encountered? 

(a) Were craftsmen able to install, test, troubleshoot, or 
was system so foreign that engineers were needed? 

(b) Was elaborate/non-routine test equipnent needed? 

2. Are there any changes to system design that, in retrospect, 
would enhance or otherwise fa\'orably influence initial field 
deployment and testing? 

E. System Operations Experience 

1. Has system operated as expected? If not, What has arisen 
that was unexpected? 

2. Is performance adequate in retrospect? 

3. Is reliability adequate in retrospect? Is redundancy used? 

4. Is naintainability adequate in retrospect? What have been 
MITR experiences? 

5. Is flexibility for grcwth/change acceptable in retrospect? 
Are any future requirements now foreseen that would have 
influenced design? 

F. Any Other Information Useful to U.S. Govemment/DoD in Planning 
for the Defense Comnunications System (DCS)? 

WESTERN UNICgJ DIGITAL NETWORK 

The Western Union Telegraph Conpany network in this survey was op- 
erated as a testbed until it was dismantled in late 1973 or early 1974. 
Ihe system utilized line-of-sight microwave as the basic trunking med- 
ian and provided point-to-point data services. The system, vdiile only 
deployed over a limited area (Cincinnati to Atlanta) was conceived and 
designed to be eventually expanded on a nationwide basis. 

The system utilized pcurtial response digital signals in the lower 
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baseband portion of an existing analog microwave radio system. Simul- 
taneously, 600 ETM voice channels occupied the upper baseband. 

The system was originally planned and designed around a five-level 
time division multiplex hierarchy. The fifth level of multiplexing 
(20 or 40 Mb/s output) was never inplemented. 

Both the planned ard inplomented systems eirployed a hybrid synchro- 
nizing arrangement wherein the first (or lower) twD levels of multi- 
plexing were synchronous system-wide while the top twD (or three) lev- 
els of multiplexing employed pulse stuffing. This design, using asyn- 
chronous operation for the higher level multiplexers, minimized the use 
of high speed elastic stores, or buffers. Path length changes and oth- 
er multi-bit delay variations were taken care of by the pulse-stuffing 
capability. Thus, the network was a hybrid, i.e., it was not purely 
synchronous or asynchronous. 

The timing and synchronization system likewise was a hj^arid in that 
it was conceived as a mixture of both master-slave and independent 
naster types of systems. The segment of the planned system which was 
deployed between Atlanta and Cincinnati was basically operated as a 
master-slave system, but had the hybrid capability. 

The timing and synchronization subsystem of the Western Union net- 
work was built around a set of redundant Disciplined Oscillators and a 
corresponding Interface Unit. The Disciplined Oscillators are basi- 
cally phaselocked loops (PLL's) and the Interface Unit is a control 
circuit viiich selects the reference source to v^ich the PLL's are 
locked. 

For master-slave operation of the network, a highly stable oscilla- 
tor such as a rubidium standard is fed into the Interface Unit at the 
master station. All other station clocks in the network then receive 
their reference from one or more of the 56 kb/s synchronous data chan- 
nels f^anating down from this master. At these slaved stations, the 
Interface Unit selects which 56 kb/s signal is to be used as the re- 
ference. 

For independent master operation, each node in the system where ti- 
ming is derived has a Loran-C receiver subsystem. This subsystem re- 
ceives a ground wave signal from the Loran-C syston and integrates it 
for a long period to provide a highly stable 1 MHZ signal to the In- 
terface Unit and thence to the Disciplined Oscillators. All stations 
deriving timing from the Loran-C system are thus synchronized. 

The Disciplined Oscillators are the heart of the Western Union ti- 
ming and synchronization system. They were designed using a voltage- 
controlled crystal oscillator and a third-order feedback loop with 



three time cxmstants. The third time constant of approximately 200,000 
seconds provides a hiighly stable output for several days in absence of 
an input reference. The third order loop remembers the rate of change 
of the difference in frequency between the internal VCXO and the refer- 
ence frequency. When the input signal is removed, the feedback loop 
continues to correct for this difference. Thus, if the reference has 
renained unchanged, when it is reapplied, the drift rate of the VCXO 
will have been ccnpensated as if the reference had not been lost. The 
other two time constants are 50,000 seconds for the second loop and 
(switch selectable) 5, 25, or 500 seconds for the first loop. 

Because the Western Union system was never made fully operational, 
seme of the questions asked were not directly answerable as noted below. 
A. Fundainental Reasons for Choosing WU T/S System as Inplemented 

There were several cxiteria used in selecting the Western Union sys- 
tem as inplemented. Cost was a primary consideration in choosing a 
synchronous system in that it was felt that the relatively large num- 
bers of lower level multiplexers could be much sinpler. The reason for 
choosing asynchronous higher level mul^-iplexers was to avoid the re- 
latively large costs of high speed elastic stores. A synchronous mode 
for the lower level multiplexers would also allow switched circuit op- 
eration, and this was the prinrary reason for selecting a synchronous 
network. 

The master-slave technique, augmented by an independent master cap- 
ability, was chosen for its sinplicity of design and operation. At the 
time, it appeared that this technique, along with independent clocks, 
was well within the state of the art and achievable in practice, vdiere- 
as systems such as mutual synchronization were considered (by Western 
Union) as unproven and questionable. 

The independent clocks system was also considered, but it was con- 
cluded that the cost of avoiding periodically overflowing buffers was 
low enough to be well worthsdiile. Programmed, or scheduled, error 
bursts associated with resetting buffers was judged lonpalatable from a 
marketing point of view. 

The final system choice was made on the basis of engineering judg- 
ment about practical aspects and associated technical risks. Implicit 
in this judgment was a performance, cost, and maintainability assess- 
ment. Simulations were not used. 

SoTiE brief consideration of using the network T/S system to distri- 
bute precise time was given. It was judged, however, that no market 
existed to justify the espense. Being able to synchronize the system 
to another system was judged desirable and was a factor in choosing 
the master-slave technique. It was felt at Western IMion that eventu- 
ally it would be desirable to synchronize with (slave to) other U.S. 



ccxmon carriers. 


B. Theorectical Design of WU T/S Systan , 

Several inportant aspects were considered when specifying the Wes- 
tern Union T/S systan. It was initially planned to use a data signal- 
below-analog technique on existing analog radio systans and/or group 
band modans and/or leased T-1 facilities as the transmission media be- 
tween nodes. Thus the system design, including the T/S equipnent, had 
to be conpatible with these transmission methods. The capability of 
the Interface Unit to accept 56 KHz, 1.544 MHz aixl 1 MHz reference sig- 
nals reflects this requirement. 

Another inportant criterion in specifying the T/S systan was the 
required mean time to loss of bit count integrity at a node wfien the 
source of timing reference was lost. It was felt that bit count inte- 
grity should be maintained for up to several days if possible. To pro- 
vide this, a stability of one part in 10^ per day was chosen as the de- 
sign goal. 

Cost was also an inportant design criterion. In conjunction with 
the necessary reliability it was decided that the master-slave sys- 
tan augmented with a back-up Loran-C receiver/nonitor was the most cost 
effective way to provide station timing throughout the network. An 
overall goal of Western Union was to develop the digital system to per- 
mit service at rates below the corresponding analog derived channels. 
The station clock system cost approximately $25,000 in 1972 dollars 
which was felt to be conpatible with overall cost goals. 

The following specifications were used for the development of the 
Western Union T/S equipment. 

a. Stability - long/short term. 

Design goal of tl x 10~^ hours 

b. Reliability - no reliability numbers such as IvrrBF were used. 
However, Western Union engineers felt that the T/S equipment 
should be of such reliability that overall system reliability 
would be determined by the radio system's MTBF and propogation 
path availability. 

c. Jitter Reduction - no jitter reduction specifications were 
given. 

d. Cost Goals - an approximate capital expenditure goal of 
$25,000 per station clock was used , not including spares pro- 
visioning or maintenance costs. 

e. 0/M Aspects - no MTTR or other maintenance related goals ware 
specified. 

The design of the signal/clock waveforms between the T/S system 
and the multiplexers and between the various multiplexer levels was 
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driven principally by the desire to be ocrpatible with existing stan- 
dards. In most cases, the standards were not formally specified in the 
early 70 's but were de facto standards as perceived by the Western 
Union engineering staff. This basically led to using the T--1 bipolar 
format without separate clock signals wherever possible in the system. 

C. WU Practical Design/Engineering Considerations 

The only unusual or unforeseen ctostacles encountered in the develop- 
ment of the Western Union system did not relate specifically to the T/S 
system. In the older radio system used between Atlanta and Cincinnati, 
some of the klystrons used as transmitter oscillators and as receiver 
local oscillators were microphonic. Mechanical vibrations were trans- 
lated to electrical noise on the baseband in the lower frequency spec- 
trum, vdiich caused poor performance of the digital system. It was also 
found that phase and anplitude linearity of the radio system v;ere criti- 
cal, especially as tlie 600 channel voice system above the digital sig- 
nal (on the basebcoid) was loaded with traffic. If the phase/anplitude 
characteristics were not very well aligned, intermodulation products 
frcm the voice signals fell in the baseband used by the digital signal 
and degraded performance. 

All specifications on the T/S system were met in the design of the 
equipiient. It is not known what margin was achieved, i.e., how much 
performance exceeded specification. 

D. WU Initial Field Deployment, Testing, O&M 

unusual problems were encountered in field deployment other than 
those of the microphonic klystrcns and the exacting phase/linearity 
requirements. The systar was beinq operated as a test-bed, and engi- 
neers were used extensively to put it into operation. NO legitimate 
measure of hew well O&M craftsmen or other lower skill-level personnel 
could handle the system was obtained. 

Digital test equipment was not widely available at the time, espe- 
cially bit error rate test equipment, but these problems were not pe- 
culiar to the T/S system. NOn-routine test equipment was not required 
for the T/S equipment, and in today's environment, one would not expect 
bit error rate equipment to present a problem. 

In retrospect, no changes to the T/S system design vdiich would en- 
hance or otherwise favorably influence the field deployment and testing 
are known by the Western Union people. 

E . WU System Operatiais Experience 

The Western Union system did not go into regular ccrtmercial service 
so questions about operational ej^rience generally are not answerable. 
Based on the several months of field trials, the system was judged to 
be performing as 'ajqxcted. Technical pjerformance, reliability, and 
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maintainability are felt to have been as designed and no readily obvious 
change would be apparent. No flexibility for growth or change require- 
ments have been identified. When queried about requirements now fore- 
seen that would have influenced design, the Western Union engineers re- 
sponded that the deployment and growth of AT&T's Digital Data System 
{WS) would probably influence any such systan they would design today. 
For interoperability reasons, and for cost/siraplicity considerations, 
a design today would probably take a timing/synchronization reference 
signal from DDS at one or more locations and then distribviie this over 
the Western Union network in the master-slave fashion. The Loran-C 
capability would not likely be inplanented. 

F. Other Information Useful to DoD Planning of DCS 

The Western Union digital systems design/deployment ejqserience did 
not result in specific T/S information other than that described above. 
An opinion was ventured by the Western Union engineers with regard to 
overall digital systan, however : if possible, DoD would be well advised 
to avoid trying a hybrid system such as the digital MUX's on cin analog 
radio. They feel that the most cost-effective, least troublesome route 
to follow today would be to initiate any such system as a pure digital 
network, and not attenpt to incorporate existing hardware. 

CANADIAN DATAROUTE 

The Trans-Canada Telephone System (TCTS) is an association of eight 
of the largest telecormunications cortpanies in Canada. Within TCTS, 
the Computer Ccxtinunication Groip (CCG) is dedicated to data conmunica- 
tions services. After initial experimental work in 1971 with a syn- 
chronous digital network, COG announced the development of a much more 
expanded synchronous digital network called the Dataroute. The system 
provides point-to-point service to fifteen metropolitan areas in Cana- 
da. It is presently in service carrying customer traffic. Its timing 
and synchronization system is basically a variation of the master-slave 
technique. 

The Dataroute system was designed to utilize existing long haul 
facilities of the manber cdmon carriers of TCTS. Most of the long 
haul facilities of these carriers are 5 GHz microwave radio systsns. 
Group band (12 equivalent 4 kHz voice channels) modems are the basic 
method eirployed in the Datarouue system to utilize these microwave fac- 
ilities. These modems, used over the analog system, provide a 56 kb/s 
channel vhich is the basic trunk for the digital system. The modem 
typically interfaces with group level equipment in a FDM carrier sys- 
tem. 


An alternate way of providing 56 kb/s trunks for Dataroute is by 
use of a Time Slot Access Unit on T1 facilities. This equipment con- 
verts the 56 kb/s signal to a 64 kb/s data stream suitable for inser- 
tion directly into a PCM voice channel. This type trunking is used in 



Dataroute where T1 facilities are available. 


Channelizing for £)ataroute is provided by a two- level time division 
nultiplex system. The highest level multiplexer is bit interleaved cind 
synchronous on both the input and output. It provides channels for all 
circuits of 2400 b/s or greater speeds. It can be prograntned for any 
ocmbination of iiput channel speeds vdiich arc multiples of 200 b/s. The 
multiplexer in conjunction with the T/S subsystem minimizes delay 
through the network by time aligning all transmit and receive frames at 
a node. This capability bo align the transmit and receive frames al- 
lov.'s channels to contain sutmulciplexed channels v4iich need not be de- 
multiplexed as the carposite channel is patched through a node on a 
drop-and-insert basis. This is achieved without adding framing over- 
head to multiplex the subchannels. This time alignment also allows 
patching data signals only (no clock) between multiplexers. 

The second level multiplexer is nomally progranmed to accept both 
a bit clock and a frame clock from an external source (the T/S system) . 
Ihis configuration is used at nodes where a ntation clock is used - 
normally wherever two or more links come together. However, at termi- 
nal sites which connect to a single other location, the multiplexer 
can be progranmed to slave to a bit clock and frane clock derived di- 
rectly from the inocming line. 

The lower or first level multiplexers used in Dataroute are char- 
acter interleaved machines vdiich provide asynchronous low speed chan- 
nels. The high speed sides of these multiplexers are operated synchro- 
nously and feed into the low speed ports of the synchronous second le- 
vel multiplexers described above. 

From the beginning of the TCTS data caimunications network, the 
T/S system has evolved through three stages. Originally, a very siiiple 
naster-slave system was 'ised with a rubidium master located in Toronto. 
The next stage, and that vdiich is primarily in operation new is a 
master-slave augmented with extra capabilities. This T/S system has 
been in operation since early 1974 in conjunction with the siabsystems 
previously described and today forms the heart of the Dataroute. Data- 
route people refer to the augmented master-slave system as Hierarchical 
Master Slave (HMS) . 

Plans are presently underway in the COG to add a third level of 
multiplex to the Dataroute and with it to overlay a higher level T/S 
subsystem. This "new" T/S network will again be essentially a pure 
naster-slave type which will obtain its reference from the same basic 
source as the present HMS system. A new third level multiplexer and 
the new T/S system will operate at 1.544 Mb/s and will provide channels 
for the existing 56 kb/s Dataroute. The two timing systems (new MS and 
old HMS) will organize and run independently except for the cermon 
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master reference. 


The new higher level MUX and T/S equipments are presently in the 
early field trial and deployment stages and are not discussed further 
in this report. 

The HMS system is discussed in what follows and is referred to sim- 
ply as the Datciroute T/S system. 

The Dataroute T/S system is implemented by a master-slave system 
in vMch double endedness and self-organization are incorporated. 
(Directed control is of course used as it is inherent in master-slave.) 

( The HMS system is like a conventional mster-slave system in that 

netv^’^rk timing emanates from a stable master S50urce located scmevdiere 
near the geographical center of the network (Toronto in this case) . 

Ihis timing is fed down the network in tree-like fashion. Self-organ- 
ization is acconplished in a hierarchical fashion hence the HMS name. 
Accotipanying the timing signal is a ranJcing, or figjre of merit signa- 
ture vdTOse value at ainy location is generally dependent on how remote 
the location is from the master source. This signature is carried on 
a 400 b/s overhead channel. The signature is carried as a three digit 
number where digit one designates the node from vhich the clock first 
originates. Digit two signifies the number of links that have beei 
traversed from the node having the original (master) clock, and the 
third digit carries the value of the immediately prior node. As tfie 
timing reaches each node and is passed on, the last two digits are up- 
dated. At every node in the neta>?ork each incoming 56 Jda/s stream car- 
ries such a signature. At a node with several incoming channels, the 
stream with the lowest valued signature is chosen as the one from vhich 
to derive timing reference. Continued monitoring will pennit the net- 
vork to automatically reconfigure in event of loss of master or a link 
failure; timing loops will not be set up in the reconfiguration. Auto- 
matic reconfiguration vathout setting up unstable timing loops vras the 
primary reason for using the HMS system as opposed to a straight MS 
sys't'em. 

Another system feature incorporated in Dataroute is called Master 
Frame (MF) . This concept is basically the dissemination of a framing 
epoch marker by the timing supply at each node in addition to a bit 
epoch marker. In the 56 kb/s system, the basic frame is 280 b±s 
long, and the frame ep jn occurs at a 200 Hz rate. Thus the nodal tirfi- 
ing supply, or station clock, delivers to each multiplexer both a bit 
clock at 56 KHz and a frame clock at 200 Hz. This deliveren frame 
clock is the Master Frame. This carmon frame marker is obviously time 
aligned at all multiplexers in a given station and is useful in mini- 
mizing delay through the netwc>rk for a channel patuned through a sta- 
tion. This minimization ccmes about because the low speed input buffer 
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reqiJxanents are minimized on each la'ltiplexer. The Master Frame con- 
cept also allows drop and insert of sutmultiplexed channels without re- 
quiring overhead framing bits fcr the :^ahmultiplexing. 

Another feature called UniversaJ Time Frame is inplfinenteu in the 
Dataroute T/S system. Universal Time Frame (UTF) is an embodiment of 
double endedness. This embodiment actually involves an inter function- 
ing of the T/S subsystem with the secoi level multiplexers (56 kb/s) , 
The designers of Dataroute included UTF to further minimize network 
user delays. They recognized that this feature would also allow pre- 
cise time dissenination; hov^^ver, it i'" not exploited in the system. 

Universal Time Frame operates as follcvs. Consider the case vdiere 
one of two cxximunicating nodes is slaved to the other for tluijig diss- 
onination purposes. The two multiplexers when initially put into ser- 
vice, will be prograitmed such that each uransmitted frame is advanced 
in time (relative to local master frame) an amount \-ihich slightly 
exceeds the propagation d'lay. Tiiis delay is knc^.n from theoretical 
or empirical (or both) considerations. Because the frames are advanced 
at the transmitter, they arrive at the far end of the link (nearly) in 
phase with master frame. The receiving multiplexer at the master sta- 
tion of the (xntnunicating pair measures the time alignment of the re- 
ceived frame by measuring the operating position of its high speed in- 
put buffer. This buffei position is tfien ccrmunicated to the slaved 
node via the overhead channel previously mentioned (with regard to die 
HMS signature information) . At the slaved node , this inf ontation is 
used to generate an error signal to the local station clock. This sig- 
nal causes the clock to adjust its frequency so as to advance or retard 
tne time alignment of the frame marker. Correcting the slaved clock 
obviously adjusts its master frame epoch relative to the master sta- 
tion. This continuing, (^miamic correction of the slaved station's mas- 
ter frame obviously serves to keep tlie slaved frame marker time aligned 
with the master station marker, and the alignment passes down through 
the netovork. 

The transmittal of buffer position information to the other node 
constitutes a form of double endedness. 

Specific answers to the set of questions follow: 

A. Fundamental Reasons for Choosing Dataroute T/S System as Inplemented 

Efficiency (ratio of data bits to data-pli-^s-overhead bits) , ease of 
sutmultiplexing many different rates of user channels, and system sim- 
plicity were the chief reasons for choosing a synchroiious digital net- 
work. The criteria for choice of vdiich type synchronizing system were 
largely based on the Canadian network topology. It was envisioned that 
a long thin network would initially be deployed but that a highly in- 
terconnected topology would evolve. Flexibility to accemmodate rela- 



tively unpredictable growth was also c»nsidered essential. 

Ihe Dataroute T/S system is master-slave augmented by double ended- 
ness and self-organization. A straight master-slave system as well as 
a mutual synchronization ^stem were also considered. The choice was 
made on grounds of enhanced security, ease of network expansion, and 
relative insensitivity to link failures. The choioe was cilso tendered 
by a desire to retain sinplicity to the extent possible. Thus perfor- 
nance was directly factored into the choice; costs and 0/M influences 
vere indirectly factored in. 

Referencing the system directly to UTC or to a national standard was 
not considered as needed for Dataroute, but it was recognized that the 
Uhiversal Time Frame feature allows distribution of (relatively) pre- 
cise time. CCG did not foresee a need or market for this service to 
the extent that the additional hardware would have been cost justified. 
On the other hand, ooirpatibixity with other systens was felt to be a 
future need worthy of factoring into the choice of T/S system. It was 
e;q«cted that interfacing with other synchronous digital networks would 
be needed and it was recognized that a master-slave type system allows 
this sinply by using a oonmon reference as the primary master source. 

B. Theoretical Desi g n of Chosen Dataroute T/S System 

After the choice was made as to the type of T/S system, a nunter of 
factors constrained the design of the hardware. This basic trunking 
medium was primarily to be over existing 4 C3lz microwave systems and 
the design had to account for the characteristics of this transmission 
channel. The rate of delay variations due to the transmission media, 
the induced jitter on the timing signals, the probability of outages 
and link failtires, and the probability/frequency of re-routing of 
transmission channels were among the factors considered. 

Othe~ factors vdiich led to specifications on the T/S hardware in- 
due.- d: the desired time to loss of bit count integrity in case of 
failures in the timing dissemination chain, the reliability budget for 
the overall system, and cost goals throughout the system. 

C. Practical Cesign/Engineering Qonsiderations for Dataroute 

No significant obstacles were encountered in the design 'developnent 
of the Dataroute T/S system hardware. Each node in the system contains 
two station clocks. Basically these clocks are made of a highly 
stable vexo configured in a phase- locked loop. The loop is designed 
with a very narrow bandwidth (nominally specified in microhertz) and 
with variable slev^-rate control. Each clock also contains the logic 
circuitry' to accept and cotpare up to 32 inceming reference signals 
plus signat’jres and the logic circuitry to conpute its own outgoing 
signature. Additionally of course, there is digital frequency syn- 
thesizer circuit"y for deriving the various clock and frame rate sig- 
nals. 



None of the above circuitry presented inordinarily difficult design 
prx±)lQns. Hie logic speeds involved were not extreme and the usual pre- 
cautions in board layout, wiring procedures, etc. , were sufficient to 
ensure good performance. 

D. Initial Field Deployment, Testing, O&M 

Specific, identifiable installation problems were not encountered 
in the early field installation of Dataroute. As is described later, 
some difficiilties were met that can generally be traced partially to 
the ccrplexity of the method of iitplanenting the double ended T/S sys- 
tem concept. These are discussed mere fully belcw under Systems Oper-'- 
tions Experience. 

Because the deployment of Dataroute vas planned to be carried out 
as a field trial, engineering- level personnel fron both OCG and hard- 
ware contractor organizations were used extensively. The fact that the 
field craftsmen from the constitutent TCTS organizations were mosfy 
ejperienced in FDM-FM analog technology influenced the decisicxi to rely 
heavily on engineers for initial deployment. 

NO specialized or otherwise non-routine test equipments vere re- 
quired in the early field deployment of Dataroute. As with most digi- 
tal ^sterns, the indispensable tool utilized was an oscillosccpe that 
could be externally synchronized. A logic analyzer type instrument 
designed to interface with the multiplexers, was used and is of course 
a specialized item. This however, was not necessarily required, i.e., 
the system could have been put into service without its use. More im- 
portantly, the use of this instrvment was not influenced by the type of 
T/S system in use. The utility would have been the same with indepen- 
dent clocks, mutual synchronization, or whatever at the T/S rrEthod. 

E. Systems Operation Experience 

The Dataroute t/s system has operated as ejpected. Basic specifi- 
cations have been met and no problems directly related to concepts or 
theory have been encountered. However, two issues have arisen. One of 
these relates to a practical problem on clock distribution in the ter- 
minal. The other relates more to system design philosophy. 

The clock (and frame) signals in a Dataroute node are distributed 
from the station clock equipment raede to each rack of multiplexing 
equipnent ty means of a redundant bussing scheme. TWo busses, one from 
each of the (redundant) station clocks, distribute signals to each rack 
of multiplex equipment. This distribution is accomplished via distri- 
bution modules mounted on the tc^ of each bay (rack.) of multiplex 
equipment. Several prcbletis have arisen with this. It was found that 
although i:p to six multiplexers could be driven in tandon, this 
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was insufficient in sane nodes and further e:q>ansion was not easily im- 
plemented. The lesson in this is to plan chat the clock distribution 
system be easily expanded and that ideally, the ejq^ansion capability be 
unlimited in size. 

Initially the clock signal was a current source- to-ground format. 
This contributed to the limitations on the number of multiplexers to be 
driven. It also gave rise to interference problems through noise pick- 
tp, and oornnon mode coupling. This problem was eliminated by redesign- 
ing with a format using a traasformer-coupled. bipolar, balanced signal 
of several volts. 

The second (philosophical) issue relates to the capability of the 
T/S system. As has been described, the Dataroute T/S system is basic- 
ally a master-slave with three cotplementing features. 

1. Hierarchial Master-Slave (self-organizing) 

2. Master Frame clock along with bit clock 

3. Universal Time Frame - a form of double eidedness 

In Dataroute, the reason for using Hierarchial Master Slave was to 
allow automatic reconfiguring of the network, without setting up unsta- 
ble closed timing paths, in the event of link or master clock failures. 
However, experience has shown that network reconfigurations are so rare 
that the utility of HMS is questionable. In other v^O)rds, the self-or- 
ganizing feature is so seldom used that a cost/benefit analysis based 
on this experience would likely dictate not incorporating it in the de- 
sign if the choice were to be made again for Dataroute. 

Note: The rarity of failures has other implications for military 

systems. Because of the rarity, it is possible, even probable that 
personnel may be so "out-of-practice" that they cannot diagnose and 
oorrect T/S system problems in a timely manner and may actually com- 
pound the problem through incorrect activities. This of course oould 
be disastrous in a wartime situation. Thus automatic self-oorrection 
may be much more desirable. 

The second feature. Master Frame, was iirplamented to allow cross 
patching between multiplexers without the need for (low speed channel) 
input buffers. In retrospect, the Dataroute operators now question 
whether or not the ability to- dispense with the low speed buffers is a 
significant advantage. With LSI and VLSI usage growing exponentially, 
the cost of adding buffering to channel cards is not large. The sim- 
plicity of channel cards is a positive feature however, and the ability 
to patch channels with only signal leads (i.e., without se^ ^ate clock 
signals) is a positive feature. This reduces the problems that arise 
when large nunhers of channels in a single rack make physical room for 
the signal and clock cables scarce and the opportunity for wiring err- 



013 are catpounded. 

The third significant featuie in the Dataroute T/S system is Univer- 
sal Time Frame (UTF) . The original reason for UTF was to minimize user 
delay through the network j UTF does this by assuring that the high 
speed frame alignment buffers in the synchronous multiplexers remain, 
on average and in absence of propagation delay variations, at their 
center or null positions. However, with the hindsight that has accrued 
through several years of operating experience, it is new felt by the 
Dataroute operators that user delay through the netvirork is not criti- 
cally inportant. This is increasingly true as data users begin to move 
away from protocols that orploy acknowledgement type transmissions 
(such as Binary Synchronous protocol) . Delay considerations are also 
less critical to users vdio continue to erploy acknowledgement protocols 
but vho take advcintage of the better channels (lower error rates) in- 
creasingly avciilable. Longer data block sizes are feasible (to a point) 
on the better channels and with the longer blocks/fewer errors, turn- 
around time, or delay, is less degrading to overall throughout. 

The reliability of the T/S system on Dataroute has proven to be de- 
graded by the corplexity of the station clock. The HMS function plus 
UTF are inplarented at the cost of increased catplexity. This increased 
oexplexity serves to cause increased hardware failures plus contributes 
to maintainability problems as discussed below. 

The maintainability of the T/S system on Dataroute has proven to be 
significantly more of a problan than anticipated. This is partially 
due to the catplexity of the double ended, self-organizing system and 
partially due to the method of inplementation. The system has been in 
operation over 5 years now and maintenance craftsmen do not yet totally 
catprehend the synchronizing hardware. Engineering level personnel are 
heavily involved in T/S system maintenance and operation. Even people 
with this level of training/coipetence lack in understanding the sys- 
tan. This catplexity and the attendant lack of familiarity by the 
field people serves to reinforce the skeptical attitudes as to the cost 
benefits of UTF and HMS. Considerable effort has been expended to 
write and rewrite equipment manuals and to upgrade training, but as a 
practical matter, the maintenance people continue to regard the T/S 
equiprent with considerable confusion. Much of these corprehension 
difficulties can be attributed to the specific method of inplementing 
the double endedness. If, for exanple, the information passed from 
one end of the link to the other were available to maintenance/opera- 
tion personnel in familiar units of time (seconds or microseconds) , 
much of the difficulty would probably be overcone, (Note that the in- 
formation is presented as a dirmensionless number referred to as "Diff" 
in the present inplementation) . 

The flexibility of the Dataroute T/S system is judged to be accep- 
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table. No restricticais on growth or change to the network are iirposed 
by the synchronizing system. It is notable however that in the planning 
for the 1.544 Mb/s network, which is a growth step for Dataroute, a 
strai^t master-slave technique for T/S is planned. Future requirenents 
appear to be for a longer, less interoonnected netvgork - one wherein 
closed loops are less probable, 

Mvice to be offered to U.S. Goverrment/DoD in planning for T/S of 
the future DCS was succinct: sinplicity should be watchword. To the 

extent that survivability, security, and other specialized military con- 
siderations allow, the simpler the T/S system, the better will be the 
performance. Maintainability will be a strong function of simplicity 
and reliability/availability will probafcly re related to maintainability 
more than any other system parameter. 

I ftTRAN DIGITAL SYSTEM 

Data Transmiission Gorpany (Datran) was a specialized oatimon carrier 
OQRiany formed in the late 60's to design, build, and operate a nation- 
wic‘ network for data ocninunications. 

The system designed and built ty Datran is distinct from the others 
surveyed in this r^xort because the total network v^s newly conceived, 
designed, and built as a digital system. Ihere ware no constraints of 
having to utilize existing plant or equipment; nor was the design 
forced into compromise in order to have a voice channel or other analog 
transmission capability. It was therefore feasible to consider tech- 
niques, including the T/S system, vduch perhaps ware not feasible as 
candidates for the other systems surveyed. 

The radio equipment in the Datran network is a digitally modulated, 
line-of-sight micrcwave systan which uses 8-level PSK modiilation. Fully 
loaded, it acc^ts two, phase-synchronous 21.504 Mb/s data signals as 
well as a coherent 21.504 MHz clock signal. 

The timing signal is recovered and the data regenerated at every re- 
peater in the Datran system. Timing recovery is accomplished in 
14.7456 MHz phase-locked locps. The loop bandwidth (3 dB) is 1 KHz in 
the radio systan. The free running stability of the oscillators in the 
tuning recovery loops is IzlO" (short term). 

A three level multiplexing systan is used in the Datran network. 

All three Datran multiplexers are synchronous machines. Each utilizes 
bit interleaving to ccmbine parallel low rate channels to a higher 
speed seriad bit stream. Every multiplexer features buffers on both 
the hi^ and low speed input ports for jitter reduction and to accomo- 
date propagation delay variations. 

The systan v4iich provides timing and synchronization for the Datran 
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network is a straightforward afplication of master-slave techniques. 

The T/S hardware is primarily maie up of an equipment called the Datran 
Station Clock (DSC) which supplied bit rate clock to all levels of mul- 
tiplexers. However, the overall network timing subsystem is oonposed 
of functions in the multiplexers, the DSC's, the microwave radios (as 
a timing dissemination channel) , and one or more rubidium standards 
which provide the basic reference signal (master source) . 

The DSC basically is a highly stable, highly reliable disciplined 
oscillator with the ability to "remember" an input reference frequency 
after loss of the reference. It includes a frequency synthesizer for 
deriving various system clock rate signals from the basic oscillator 
frequency and line drivers for distributing these clock rate signals. 

It also includes logic hardware for interfacing with the multiplexers 
on a systems basis and for controlling its own functioning. The DSC 
provides clock signals at 21.504 MHz, 2.688 MHz, and 168 KHz. 

The heart of the DSC is a triplicated set of modules called Timing 
Generators. These are highly stable 's, each in a phase-locked 
loop configuration vhere the loop error voltage is simultaneously fed 
to the VCXO and also digitized, filtered to an effective 0.04 Hz band- 
width, and stored in mstiory. The digitizing is done to a 12-bit accu- 
racy vhich translates to a frequency accuracy at the VCXO of a 5 x 
10~^^. This stored error voltage is applied to the VCXO in instances 
v\hen the station clock transitions frcm slave operation to master. Be- 
caiase of the narrow bandwidth of the digitized error feedback loop, in- 
put jitter with frequency content down to much less than 1 Hz is fil- 
tered. By virtue of this jitter reduction, when the clock becomes a 
local master, frequency differences between it and the normal master 
will remain sufficiently small to maintain bit count integrity for more 
than 30 minutes (based on the buffers used in the C-MUX's) . This is in 
the presence of up to 5 percent rms jitter with a bandwidth frcm 0.1 to 
1000 Hz. 

The bandwidth of the (undigitized) error voltage in the phase locked 
loop is 4 Hz. This translates to a capture, or pull in range for the 
Slave Node of ^ 1 x 10”'. 

In the follcwing, answers are given to the set of questions as they 
pertain to the Datran Systan. 

A. Fundamental Reasons for Choosing Datran T/S System as Implemented 

The criteria used for choosing the T/S method for the network were 
principally performance, cost, and state of the art (practical state as 
opposed to theoretical) prevailing at the time of system design. Me- 
thods other than master-slave were considered. Nonsynchronous opera- 
tion using independent clocks and various types of pulse stuffing were 
evaluated as possible candidates. Synchronous techniques considered 
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included designated (or ind^)endent) master, mutual synchronization, 
and of course master-slave. Bit stuffing was ruled out as being too 
e:q3ensive to inplement in a dedicated circuit system. In a switched 
system, the cost is even more prohibitive. Independent clocks were 
discarded as a potential candidate mainly because the prospect of per- 
iodically overflowing buffers, no matter how great the period, was 
judged to be a severe marketing disadvantage. Of the types of synchro- 
nous operation considered, master-slave was chosen for cost, perform- 
ance, flexibility and sinplicity reasons. The Datran network was en- 
visioned, even far into the future, to be a topologically long thin 
system with no looped subnetworks. Because the basic transmission med- 
ium at 44 Mb/s had extreme capacity in terms of custoner channels, it 
was not considered likely that alternate routes and possible loops 
would be necessary for many years. Too, the transmission medium was to 
be designed for such reliability that the need for failure-induced al- 
ternate routing and the possibility for looped configurations was di- 
minished. Last, since transmission was to be entirely over Datran 
facilities, routing configurations could be tightly ccntrolled. All 
these reasons enhanced the master-slave choice, without self-organizing 
ccnplications . 

Sinplicity of hardware and the attendant advantages in maintainabi- 
lity and reliability also weighed in favor of master-slave as ccnpared 
to frequency averaging (mutual) . Independent masters appeared to be a 
viable choice as regards sirrplicity and cost (assuing an existing sys- 
ten such as loran-C as the basic master source) . However, the degree 
of dependency on a system vdiose operational reliability and integrity 
lay outside the Datran operation was considered a negative aspect. And 
significantly, in che case of Loran-C, it was felt that the availabili- 
ty of ground wave coverage was lacking for the total geographical area 
Datran hoped to eventually serve. 

Simulation was not used in choosing the Datran T/S system. Formal 
performance and cost analyses did influence the decision; however, en- 
gineering judgments on practicality and probability of successful, on- 
schedule design also weighed heavily. 

Systan reference to UTC or otlier standards was not considered a 
strong factor in evaluating T/S techniques. It was recognized that 
master-slave afforded relatively easy interfacing with one other syn- 
chronous system (by utilizing either a ccanmon reference source or by 
havi.ng the Datran network be a sinple extension of the other network) , 
but again this was not a strong factor in the T/S choice. 

B. Theoretical Design of Chosen T/S Systan 

The Datran systan was designed "starting from scratch" and choices 
were not constrained by existing plant o'' facilities. LOS microwave 
transmission media was an early choice. The yearly (or longer) and 
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daily delay variations over multihop, multiclimatic micxowave chains 
was not precisely known, and the T/S system chosen would have to be ver- 
satile enough to handle this unknown. 

The following specifications were design goals for the Datran T/S 
systerr.: 

a. Stability (without primary reference ) 

;J; 1 X 10'^ per day 

+ —8 

_ 4.5 X 10 per 6 months 

With a jitterless primary reference iiput, the T/S station 
clock was specified to have less than 1 percent rms output 
jitter. 

b. Reliability 

The station clocks in the Datran network were specified to 
have a mean time between catastrophic failure of greater 
than 1,613,000 hours. 

c. Jitter Reduction 

The station clocks were specified to operate satisfactorily 
over up to 33 tandon microwave links. To acconplish this, 
a jitter reduction of at least 6 to 1 (for iiput jitter 
contained in a 0.1 to 1000 Hz bandwidth) was specified. 

d. Cost Goals 

In 1974 dollars, cost goals in the range of $50K per (ful- 
ly configured) station clock were established. 

e. 0/H Aspects 

Guidelines on modularity, nun±>er and nature of visible 
a].arms and indicators, and diagnostic aids were established. 
A mean-time-to-repair (after reaching equipnent site) of 
less than 30 minutes was established as a design goal. 

The principal criterion was perfonrance as opposed to conformity 
with existing standards. This was especially true for interfaces be- 
tween multiplexers and between multiplexers and the radio. At custoner 
interfaces, confonnity with widely used standards was necessary. At all 
interfaces in the Datran network, the timing signal is carrivjd on a 
separate conductor from the data signal. This contrasts with the poss- 
ibility of having the clock signal not be distinct from the data sig- 
nal as is done for instance in the familiar bij- olar of Tl systans. 

C. Practical Design/Engineering Considerations in Datran T/S 

Few notably difficult or unexpected obstacles were encountered .’n 
the design and development of the Datran T/S hardware. The design of 
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majority voting logic for controlling the triplicated functions in the 
DSC required close attention to signal delays, board layouts, etc. Ad- 
herence to good engineering practices common to nanoseconds speed logic 
design ensured a minimum of problerrB in this area. 

One problem was encoiuitered in a systems level test of the T/S equip- 
ments. The problem related to the ability of the DSC to properly hold 
the system frequency when a jitter cornpted reference signal vss lost. 
It was found that the frequency spectrum of the jitter on the reference 
was of critical importance in how well the system frequency was main- 
tained. The overall specification required that when using 32 bit 
buffers, bit count integrity would be assured for at least 30 minutes 
v\hen a reference with 10 percent rms jitter, with a frequency spectrum 
between 0.1 and 1000 Hz, was lost. Recall that the bandwidth of the 
loop in the DSC which held the error voltage, or "remanbered" the sys- 
tan frequency, is 0.04 Hz. However, it was found that if the jitter 
spectrum was concentrated near the lower end of the specified band- 
width, the loss of reference would scmetimes leave the ratiatibered fre- 
quency deflected enough to cause buffer overflow in less than 30 min- 
utes. If the jitter' spectrum were spread uniformly over the 0.1 -to 
1000 Hz band, or concentrated near the high end however, the specifi- 
cation was easily met. 

This problem of maintaining bit count integrity in the presence of 
low frequency jitter was never manifested in actual field operation but 
only in lab tests where jittered references were synthesized. 

With -tlie exception of the above described phencmenon, all specifi- 
cations on the T/S system were met or exceeded by -the completed design. 
For exanple, a jitter reduction ratio of greater than 20 to 1 was mea- 
sured in the field for the DSC whereas the specification was 6 to 1. A 
mean time between maintenance repairs of greatea: -than 22,000 hours was 
also measured in the field on the T/S equipment. This caipares to a 
design calculation of 6700 hours. As for mean time to catastrophic 
failure, no such failures had been encountered at last check so that a 
meaningful quantitative s-tatement cannot be made. 

D. Datran Initial Field Deployment, Testing, O&M 

In the initial deployment and operation of the Datran netwDrk, all 
field personnel were given cotprehensive, but extranely time-cotpressed 
training on all equipiients. Ety and large, the persorinel were experi- 
enced in FDM-FK analog systems as opposed to digital systems; thus many 
of them encountered and progressed through a distinct learning period 
on the entire network. While the T/S subsystem presented no more pro- 
blems during -this learning period than did the other equipments, never- 
theless the nonfamiliarity was of significance. The triple redundancy 
of the Station Clock added to the nonfamiliarity problems -to such an 
extent that it considerably cotplicated the understanding of the equip- 
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ment. It also greatly added to the difficulty of troubleshooting equip- 
ment malfunctions. 

Datran procured several items of test equipnent related to the T/S 
system that were nonroutine. Among these were portable rubidium stan- 
dards, frequency synthesizers, and phase jitter meters. In retrospect, 
it is apparent that these equipments were not necessary for operating 
and maintaining the network. They were useful in monitoring the net- 
work performance from the point-of-view of an extended field trial 
vhere continued engineering data was desirable. However, frcm a stand- 
point of routine, day-in, day-out system operation and preventive main- 
tenance, an oscilloscope in conjunction with built-in test functions on 
the DSC is adequate. 

No significant changes in system design are obvious in retrospect 
that wou]d enhance or otherwise favorably influence field deployment 
and testing. 

E. Datran Systems Operating Experience 

The T/S subsystem in the Datran network has operated substantially 
as expected. No significant problons have been encountered that would 
give reason to reevaluate the basic concepts or upgrade the systan. 
However, it is probable that a simpler, less ccmplex system would be 
designed if the task were to be done again. Some reasons for this are 
noted below. 

The fully triplicated, highly reliable (and relatively expensive) 

DSC was deployed by Datran at every three-way junction station and at 
every temanal. In retrospect, such reliability and expense is proba- 
bly only justified at remote three way stations. Even there, the need 
for triplication is questionable. The DSC's provide quite long mean- 
time between maintenance repairs; and the nature of failures encoun- 
tered to date indicate that double redundancy (and the simplified con- 
trol hardware inherent in this) would have sufficed. 

Likewise, experience gained in actual operations of the system ten- 
ded to indicate that early concerns over jitter on the recovered timing 
signals was not warranted. In the actually deployed Datran network, 
the longest span between DSC's was eleven microwave hops. Had the sys- 
tem been deployed to the west coast as planned, spans of over thirty 
hops would have been encountered. Nevertheless, experience with the 
eleven hop system indicated that jitter buildup was much less than ex- 
pected. The worst measurement for the eleven hop system was jitter of 
less than 4 pearcent rms which would extrapolate to much less than 10 
percent jitter on a 30 (+) hop span. 

The Datran network actually deployed was a topologically thin, long 
line network with no closed circuit configurations. Thus no opportun- 
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ity evei' arose to evaluate bit count integrity type questions cn loss of 
continuity of the timing disseminaticn diain. 

Perfontance of the Datran T/S system was judged adequate, and relia- 
bility was more than acceptable and perhaps over designed in seme appli- 
cations. 

Ihe maintainability of the T/S system probably should be judged ac- 
ceptable to slightly itarginal. The only questionable area vrould be in 
tte use of triplicated functions as noted above. This tendat to ob- 
scure craftsman understanding of the equipment and to render trouble 
shooting more difficult. It was certainly not a severe problem parti- 
cularly for the better qualified, more industrious field pecple vho 
would put forth the effort to mderstand the hardware. 

The flexibility for growth/change of the Datran Station Clock is 
judged acceptable. It was found that in one instance, added driver 
ports were needed to enable clocking additional multiplex bays in a 
station. Carefiil, tedious work was necessary to add the capability to 
an on-line DSC without disturbing customer traffic, but it was accom- 
plished. Should a redesign ever occur, it would be desirable to pro- 
vide for easier (physical) e}$>ansion of the drive c^jability; this was 
not a major problem, hewsver. 

F. Any Other Information Useful to U.S. Government DoD in Planning 

Defense Coimunications System 

The same advice regarding sinplicity of equipments as offered by 
Canadian Dataroute operators vould be evidenced by the Datran experi- 
ence. The less oorplex the system, the better will be the reliability 
and maintainability. And, in the absence of major design errors, main- 
tainability will probably iitpact performance more than any other single 
factor. 

One other comment is pertinent. Cerpared to the late sixties/early 
seventies when the Datran network was designed, there is a plethora of 
data on digital systems available now. Each succeeding conference or 
synposium on broadband oanmunications provides additional experience 
documentation (as well as theoretical work) . Notable useful exanples 
are LOS microwave delay data and multipath fading degradation e>peri- 
ence. Ctoviously, all these sources should be used in planning the fu- 
ture DSC timing and synchronization systems. 

TIMING/SYNCHRONIZATION PLANNPIG FOR NO. 4ESS 

As noted in the introd\x:tion, original plans were to include infor- 
mation on the inplementation and cperational experiences of Bell Sys- 
tem's Digital Data System in this report. (This system is a private 
line synchronous network put into operation in the U.S. in 1974). Un- 
fortunately, this information is not available; Bell officials decline 
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to provicSe such data due to the extensive pending litigation revolving 
around the Digital Data £3ysteni (EOS) . These Oj-*'icials v^re, however, 
willing to provide infonration on the current planning and engineering 
efforts related to synchronization of a network of large digital swit- 
ches presently being inplemented throughout the U.S. This information 
was provided by way of interview with several engineers currently in- 
volved in the engineering and planning efforts. In this section, this 
infoimation is described. 

The No. 4ESS is an electronic, softwere-oontrolled toll switching 
center designed to tie local facilities to the nationwide long distance 
network. The switching center is basically a digital machine vhich in- 
terfaces both digital and analog trunks. At the inception of No. 4ESS 
deployment, and at the present time, analog trunks predoninate in num- 
ber. As digital trunks becane increasingly available hcwever, (a trend 
that is growing rapidly) , the interconnected No. 4ESS systems take on 
more and more attributes of large integrated digital networks. As this 
h^jpens, f'’9 advantages of reliably synchronizing the vhole network are 
obvious. This is recognized by Bell and considerable effort has been 
expended to plan for this synchronization. Much of this planni j is 
inextricably interwoven with planning and analysis that has been going 
on at Bell Labs since the early sixties. This early planning was the 
initiation point in the discussions with Bell engineers. 

In the early sixties, various users of the Bell Telephone System be- 
gan to inplemsnt digital networks. At first, relatively elaborate word 
stuffing was used to provide a synchronizing mechanism for the networks. 
The cost (at that time) of shift registers causesd a change to bit-by- 
bit pulse stuffing techniques and eventually to consideration and ana- 
lysis of other synchronizing metlxxis. Ctonsiderable attention was fo- 
cused on the concept of mutual synchronization. Many investigators, 
both inside Bell Labs and elsewhere, studied and published papers on 
the technique. One of the advantages ascribed to mutual syrschroniza- 
tion was that it was "auiunistration free". However, it was decided 
by the No. 4ESS network planners that this is not, in reality, the 
case. As an exanple, it was noted that ip to 20,000 DS-1 (1:544 Mips) 
trunks can converge on a No. 4ESS. To choose v^ch of these would pro- 
vide the timing reference paths for a mutually synchronized system is a 
considerable prchlem. This is especially true vhen the dynamic nature 
of the topology of these trunks is considered. 

Another aspect of mutual synchronization noted as significant was 
the duTibell effect. This situation arises when two sparate, relative- 
ly ccitplex, interconnected areas are connected by a thin tru^ struc- 
ture. In this case, the input to the T/S systems in each of the ends 
of the dumbell must carry heavily veightad inportance. This tends to 
distort the survivability of the mutual sync network and to ocxrpound 
administration problems, particularly vhen the exact coposition of the 
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thin" trunking is dynamic. 


Mutual synchronization was eventually discarded as a candidate s^ti- 
chronizing method for three significant reasons. As indicat" . it was 
determined that its main attribute, i.e., administration- free operation, 
was not a practical reality. This was the priirary reason for rejecting 
the technique. A3 so, evolution in crystal oscillator technology in the 
late 60 's was a factor in moving away from mutual synchronization. Dou- 
ble oven crystal oscillators with millidegree tenperature stabilities 
became economic realities. Ihe resulting frequency stabilities added to 
the case for master-slave techniques. Additionally, digital phase- 
locked loop (DPLL) technology progressed to the point that economic 
practicality was achieved. The significance in this was that DPIL's 
with "automatic" integral-plus-prcportional feedback was then achieva- 
ble. Note that integral plus proportional feedback had long been used, 
but the integral feedback had been a mechanical screwdriver adjustment 
(to oscillator free-running frequency) . "Feedback" of course was a 
technician periodically nulling loop stress voltages. Digital inplemen- 
tation of this function meant greater autcmatic long-term stability in 
the oscillators since drift due to crystal aging could be coipensated. 

Another influencing factor on synchronization planning was that the 
DDS network began to be engineered. At first, the objective was to 
build a totally slip free network for DDS. However, the Ito. -jESS plan- 
ners took the position that a totally slip free DDS was not practically 
achievable. As an exaitple of the several reasons, most 3f which are of 
an administration/operation origin, it was pointed out that there are 
some 60,000 separate group channel trunks m the Bell System. Of these, 
an average of 20,000 yearly are dynairjx ..i. the sense that they are rr~ 
moved and subsequently returned to service for one reason or another. 
Since these trunk assets were to provide the trananission channels for 
DDS, it was felt that such a dynamic makeup would prohibit practical 
operation of a slip free network. 

For the DDS, it was eventually agreed that three system inpairments 
would be foremost. These were i) bit errors; ii) misframing vAiere mul- 
tiplexers lose frame synchronization and iii) slips where single eight- 
bit bytes are lost/gained. Additionally of course, timing jitter 
due to accumulation on the transmission lines and due to pulse stuff- 
ing/destuffing is encountered. Of these inpairments, the most signifi- 
cant in terms of timing and synchronization is slip and the byte inter- 
val defining slip. Of course, the longer the byte interval, the less 
the frequency accuracy requirements on a timing source for given slip 
rate. An upper limit is placed on byte length in the Bell System be- 
cause the buffers give rise to cross-office delay in patching 
channels. At a point, this excessive delay begins to cause echo sup- 
pressors in the circuits to hang. This led to the 8-bit byte vhicn is 
125 microseconds in a 64 kb/s channel. 
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After thG byte length was determined, an acceptable slip rate was 
evaluated. A mmiber of factors were considered by Bell in determining 
a slip rate specification. In voice channels slips are trivial. How- 
ever, for voice band data channels , slips can result in modems n train- 
ing wi—ch results in outages of ip to several seconds. Slips also can 
have a severe iitpact v^ien Cermon dhannel Interoffice Signalling is car- 
ried over a slipping channel. Ihe effect of slips of digi.tal data var- 
ies depending on the application. It was noted by Bell that ocmmercial 
applications of secure voice are foreseen over data channels in the fu- 
ture and slips could result in cryptographic sync probleiiis. 

The above and other considerations led to the Bell System eventually 
settling on a specification of one slip per 5 hours for an end-to-end 
reference circuit. This translates to a relative clock frequency dif- 
ference specification of 1.7 x 10~^. Because the possibility of longer 
frames becoming desirab’ in the future however, a ti(^ter stability 
specification of 1 x 10~'^^/day was agreed upon. 

At the present time, the working specifications cf th' timing and 
synchronization si±)systaTis for the No. 4ESS network are 

1. Clock frequency inaccuracies of less than 1 x lO^^^/day. 

2. Synchronizing unit at each switch wi.l' jdiase lock to an external 
reference. This reference \ ill be either a DS-1 signal at 1.544 
MHz or the standard Bell System Reference Frequency (BSRF) 2.048 
MHz. Note that this latter reference is presently distributed 
throughout the netjwork. 

3. The synchronizing system will be arranged in a hierarchical stru- 
cture with frequency (not timing) information emanating fron an 
atomic standard at Hillsboro, Missouri. 

4. The timing system will have a highly structured maintenance plan 
administered out of a centralized location. 

Note that specifications 2 and 3 define a master-slave system vhen 
the DS-1 reference is disseminated throu^out the network. When the 
BSRF is used, an indepervlent master system is being inple.ianted. 

It was noted by the Bell engineers that the fourth entry in the a- 
bove specification list is extremely inportant as contrasted to its 
innocuous appearance. It has historically been found that maintenance 
activities are always reflected negatively in reliability performance 
figures for ccnmunications networks. Significant percentages of out- 
ages are traceable to maintenance persorinel activities. Since the ti- 
ming sources for digital networks are so central in the operation, and 
outages eire potentially so catastrophic, it is felt that the structured 
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maintenance philosophy is critical In general, alarming and alarm re- 
action plans for the timing and synchronization hardware on No. 4ESS 
will have long time constants. A loss of reference alarm will, for ex- 
airple, be considered only a minor alarm for up to several minutes. 

Other reaction times nuntered in hours are planned. The central admin- 
istrator with network-wide statias visibility will be consiilted closely 
on all alarm responses and maintenance activities. 

The phase-locked loop in the TSU incorporates integral-plus-propor- 
tional feedback. The integrated ccnponer.t of the loop feexlback has a 
time constant of approximately 2 days. The time constant of the propor- 
tional factor is approximately 2 hours. Additionally, a rranually acti- 
vated Fast Start mode allows rapid acquisition when the TSU is first 
put into operation or followir^g a major reconfiguration or outage. Au- 
tomatic phase buildout is also designed into the TSU hardware. 

The above briefly describes many of the factors which have entered 
in the planning for No. 4ESS timing and synchronization. The Bell en- 
gineers eitphasized that many of the choices and trade-offs made were 
dominated by netwark administration considerations. This of course 
reflects the culmitiation of many years experience in operating the lar- 
gest cotmon user coitittunications network in the world. Little or no 
consideration has been given to some factors inportant to military 
strategic ccmnunications systems. Survivability in the presence of 
hostile activities is an inportant consideration in evaluating timing 
and siTichronization techniques for the DCS. Understandably, this has 
had little or no influence on the Bell designs. Likewise, interopera- 
bility with tactical systems is a consideration for the DCS not dealt 
with by Bell. 

A last topic uiscussed with the Bell engineers during the interview 
concerned the distribution of precise time. The T/S systan being en- 
gineered for tiie No. 4ESS network does not incorporate double endedness 
and thus does not readily admit of precise time dissemination. Ifow- 
ever, consideration has been given to this by Bell engineers. In gen- 
eral, the same concern for network administration considerations in- 
fluences time dissemination that influenced T/S system design. In 
fact, the Bell engineers expressed the opinion that administration pro- 
blems would so adversely effect time distribution as to make it irprac- 
tical in the Bell System. Two examples noted were network topology 
dynamics and emergency restoration. The statistic previously described 
wherein 20,000 of the 60,000 group trunks in the system are taken out 
of servdce and restored each year was pointed to as an exairple of the 
dynamics. 

Emergency restoration, and the tenpo of activities 'hat exist during 
outages was mentioned as the othe.. prominent exairple of administrative 
problori in time dissemination, i: ig outages, especially those that 


effect large cross-section trunks, inta^se effort is involved. Custo- 
mers and managanent are usually dsmaiiding quick restoral and little 
acceptance of procedures vdiich would inhibit or delay restoration would 
be anticipated. Coordination of clocks, and of new trunk path delays, 
etc. is seen by Bell engineers as just such a procedure. It was noted 
hcwevei, that autoration of such procedures use of microprocessors 
might become eoononically feasible in the future. 

StMIARy OF SURVEY 

Each of the sj's terns surveyd is a synchronous network, and each uses 
a T/S technique that is essentially of the basic master-slave category. 
The Datran network T/S systan and the planned No. 4ESS netowrk T/S are 
straightforward applications of master-slave. Ihe Western Ikiion net- 
work and particularly the Canadian Dataroute network T/S are master- 
slave with added features. 

Retention of bit-oount-integrity, i.e., the avoidance of slips was 
the foranost criteria in the design of the networks. Flexibility and 
adjudged practical state of the art in T/S technology also weighed 
heavily in the initial system designs - particularly in the case of 
the Western Union and Datran networks. Minimization of user delays 
over the network was a strong factor in the Dataroute system design; 
recognition of network administration problans stronglv influenced 
Bell System plans and choices. 

Only the Canadian Dataroute system provides for autcmatic network 
recortiguration (and autcmatic network initialization) follcwing fail- 
ure or other perturbations. Related to this is the fact that among 
the four systems surveyed, only the Dataroute requires an overhead 
channel. 

Each of the system designers recognized the crucial iitportance of 
real’Mlity in the T/S hardware/subsystem. Outages due to lack of 
timing supply are potentially catastrophic and could disrupt ve^ large 
numbers of users. Th\as, considerable effort and dollars were expended 
to build high availability into the T/S equipnent. Western Union and 
Canadian Dataroute ^stem architects acocitplished this by using double 
reflundancy with automatic switchover. The Datran i^stan utilized tri- 
ple redundancy in the station clock with all three portions contribu- 
ting in the unfailed, or normal, operating nrde. Majority voting logic 
in effect switches out the failed circuitry when problems arise. 

The field deployment experiences of the organizations surveyed were 
remarkably similar. Each of the Western Union, Dataroute, arid Datran 
initial deployment efforts were reasonably strai^tforward, but exten- 
sive use was made of engineering-levd personnel (as opposed to tech- 
nicians or craftsmen) The lack of a ready supply of digitally orien- 
ted field personnel was noted. In each case, however, the T/S system 
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type or its features had little or no influence on deployment difficul- 
ties or lack thereof. 

In the area of operations experience, only Datran and Dataroute were 
able to contribute information. ^ 'Ihe Western Union network never went 
c^jerational and the Bell System s deployment of No. 4ESS is only now 
underway. The Dataroute operational experience has been adversely ef- 
fected to some extent by the conceptual (and hardware) conplexity of 
the T/S system and to iitpletentation procedures. To date, field per- 
sonnel are not totally confortable with their knowledge and understan- 
ding of the method of timing and synchronizing the system with the ex- 
tra features of the master-slave system. Notably, the next evolution 
of the Dataroute system is planned to be an unadorned master slave. 

In the case of Datran operational experience, no significant prob- 
lotis arose which were attributable to the T/S concept. However, the 
triple redundancy in the hardware did ocnplicate craftsnan understand- 
ing and troubleshooting. Failure rates were so low as to call into 
question the need for triple as opposed to double redundancy. 

The above briefly sumnarizes the information gained in the survey of 
oonmercial digital netwarks. It should be noted that several ' ey cri- 
teria which greatly influence the design and (satisfactory) operation 
of a strategic military netwark such as the Defense Ccmnunication Sys- 
tan are absent frcm the list of connercial network criteria exemined. 
Among these are: 

1. Survivability 

2. Coipliance with Federal Standard 1002 

3. Intersystem Operability 

4. Self -reorganization 

5. Precise Time Availability 

In the above list, survivability is the most significant entry; the 
fact that a network plays a central role in national defense means that 
survivability must be as pivotal in design trade-offs as are any other 
parameters. This, of course, can oonpletely alter the conclusions of 
a cost-effectiveness study. 

Precise time availability is another feature that might be of con- 
siderable value in a defense caimunications scenario but of lesser 
inportance in the ccnitiercial world. Covert oomunications require- 
ments lead to spread spectrum and cryptographic system use. Precise 
knowledge of a coordinated time standard, or course, greatly facili- 
tates the synchronization of such systems by reducing acquisition 
search windows. Likewise, a knowledge of relatively precise time fac- 
ilitates navigation and position fixing - another distinct advantage 
in a military environment but of lesser irrportance for ccstmercial ap>- 
plications. (Although the cotmercial need for precision navigation 
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aids should not be minimized.) 

The ccsnmercial networks surveyed have made no attenpt to use their 
T/S systtans in any manner to augment or fom a basis for network moni- 
toring and evaluation. This is of course an area where the benefits 
ooiiLd be of irrportance to a catmercial system even though ic doesn't 
have the special requirarents of a military system. Such a use of the 
T/S system would generally require preplanning and enginet^ring of the 
monitoring and evaluation capability into the hardware during initial 
design, semething obviously not included in the s^^terns surveyed. Re- 
cognition of the benefits to be obtained tiirough monitoring and evalua- 
tion could influence the type of T/S system chosen. Some T/S methods 
are more ccrpatjbie and synergistic with monitoring and evaluation than 
others. 

A concluding cenment is in order. Simplicity and administration 
related issues were mentioned over and over by the systems operators. 

It was emphasized that these aspects should be central in the design of 
any T/S system. The reasoning behind these statonents is not presently 
dilatable, especially among experienced systems operators. Hoivever, 
the potential influence that microprocessor evolution is having and 
will continue to have on this aspect should not be overlooked. In- 
creasingly, it is feasible to autemate activities heretofore con- 
strained to be manually acoerplished. With automation, or machine dri- 
ven activity, the rationale behind sinplicity arguments should be re- 
examined. Without the factor of human error, much in tlie vay of auto- 
matic corrective routines are feasible. And much in the way of system 
features now considered very useful, but too cotplex to be practical, 
become realistic. This can and certainly should be factored into any 
current T/S systems design. 
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QUESTIONS AND ANSWERS 

MR. THEODORE FRENSCH, Hewlett-Packard: 

Can you elaborate on the double or triple redundancy? I can see for 
catastrophic failures that double redundancy works where, simply, 
there is no output. But what is the practicability for frequency 
shifts in double redundancy? 

MR. MITCHELL; 

That consideration is one of the things that led to triple redun- 
dancy choice in DATRAN, the idea being that Tor amplitude failures, 
certainly double redundancy allows you to decide which one has 
failed and which one is operating normally so you can switch appro- 
priately. But for phase-type changes, if you have a phase shift in 
one of two sources, it is difficult if not impossible to determine 
which one is right and which one is wrong. Whereas, if you have 
three, you can do majority voting. 

That is well and good. Our experience, again from a practical 
point of view, was the clock as it was built. Admittedly, we spent 
an appreciable sum to make it reliable. But it was so reliable that 
that came into play so infrequently, never in my experience with 
DATRAN, that that almost fell into the realm of an academic con- 
sideration. Does that answer your question or address what you are 
driving at? 

MR. FRENSCH: 

Are you saying, then, you actually don't need any redundancy or are 
you saying that catastrophic failures were more common than 
frequency shifts? 

MR. MITCHELL: 

Yes, in our case we didn't have either. When ycu have a network 
that potentially has 8,000 channels on it, you can't afford to have 
no redundancy, even if it only happens once every 30 years. First 
of all, if you had to get out and try to market the network and 
somebody started asking questions about the market service on the 
network, it would cause you problems. 

In fact, some of our station cl^,.ks in the three-way repeaters 
were 60 miles from the maintenance center. So you are talking about 
hours before the service could be restored if you didn't have some 
kind of backup there and the clock did fail. So I think you can't 
afford to go with no redundancy just from a practical point of view. 

But as far as going to the expense, and I would guess that the 
expense of the DATRAN clocks went up 50% due to the triple redun- 
dancy, and certainly the development problems were increased by at 



least 50%; you know, the design problems for the people tiiat pro- 
vided it. Had they not had the triple redundancy to contend with, 
the design would have gone a lot more smoothly; not to say that it 
didn't go relatively smoothly anyway. Yes, sir? 

DR. C. C. COSTAIN, National Research Council: 

Data Route has installed a terminal in our laboratory and we 
continuously monitor their frequencies. We are, in fact, going to 
feed back to them digitally on the network their offset, which they 
can use as they wish operationally. But I just wanted to mention 
that after we were monitoring, there was a catastrophic failure of 
their master in Toronto, or their sync channel, whatever, or some- 
thing in the loop. Montreal took over and they were rather 
pleased they had it because it was a holiday weekend with, of 
course, the least experienced, most junior engineer in charge. The 
system did exactly what it was supposed to do in case of a failure 
of equipment. We monitored it beautifully and we were rather pleased 
to have tne experiment, which you wouldn't create normally. 

MR. MITCHELL: 

Very good. You touched upon a point I neglected to mention that did 
come out in the survey. The fact that these timing and synchroniza- 
tion systems seldom fail can be important, particularly in a mili- 
tary-type environment, because it means that the people who have the 
responsibility for maintaining them when they do fail, have forgotten 
what to do because it's been so long since they had a similar prob- 
lem. So if you have a network that automatically heals it- 
self in some sense, that is an advantage. It is probably even more 
so an advantage in a military network or critical network than it 
would be in a commercial network. 



